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ABSTRACT 

Faced with concerns that students in the American 
educational system are failing to learn critical thinking, problem 
solving, and reasoning skills, and do not have the ability to apply 
these skills to real-world situations, educators and psychologists 
are researching methods of improving student learning and assessing 
student performance. In order to acquire accurate and useful 
information about student performance, classroom teachers have moved 
from traditional summative evaluation methods toward a formative 
method of assessment; teachers now have the role of 
teacher-researcher. This paper focuses on a systematic method of 
developing an authentic assessment instrument that incorporates a 
three-phase question: (l) What is to be assessed?; (2) How is it to 
be measured?; and (3) How does one observe and interpret verbal and 
non-verbal messages or data? In the first phase, teachers identify 
the purpose of assessment, recognize the uniqueness of the learners, 
and ensure that they possess a knowledge base in the relevant subject 
matter. In the second phase, teachers formulate learning outcomes and 
the criteria and standards that will be applied to them. In the third 
phase, teachers train themselves to play the role of objective 
observer. As teachers strengthen their observational skills and 
recognize their roles as t eacher-res earchers , they will ex^ci Itt * v e an 
increase in confidence and prof essional i sm, and there will be a 
direct positive effect on daily instructional and social classroom 
activities. (Contains 20 references.) (SWC) 
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THE IMPACT OF QUALITATIVE OBSERVATIONAL METHODOLOGY ON THE AUTHENTIC ASSESSMENT 

PROCESS 



Introduction 

What is taught, how it is taught, and how it is assessed are critical issues within the educational community 
today. The concerns that students in our American educational system are failing to learn critical thinking, problem- 
solving, and reasoning skills, and do not have the ability to transfer these skills to real-world situations have become the 
focus of our national educational reform agenda. In order to deal with these concerns, educators and psychologists have 
been researching various methods of both improving student learning and assessing student performance. 

In order to acquire accurate and useful information concerning student performance, teachers are turning away from 
traditional summative evaluation methods toward a formative method of assessment. This method of assessment is 
variously known as authentic assessment , alternative assessment , and performance-based assessment. These terms are 
often used interchangeably, however they are not the same. According to Marzano, Pickering, & McTighe (1993) 
authentic assessment, popularized by Grant Wiggins (1989), conveys the idea that assessment should engage students in 
applying knowledge and skills in the same way they are used in the ‘real world’ outside of school (p.13). Alternative 
assessment applies to the variety of assessments that differ from the multiple-choice, timed, one-shot approaches 
characterized by most traditional standardized and classroom evaluation methods. Finally, performance-based assessment, 
according to Marzano, Pickering, & McTighe (1993) is a broad term, encompassing many of the characteristics of both 
authentic assessment and alternative assessment (p.13). Regardless of which definition is used, formative assessment 
methods require the assessor to employ heightened observational skills to gather careful and descriptive assessment data 
on student performance. 

Observation is an important method of data gathering in authentic assessment, just as it is in qualitative research. 
The process of data gathering for the sake of scientific research has been traditionally thought of as the job of a full-time 
professional researcher. Today, however, "... research is conducted by many people in many settings” (Maniam, 1995, 
p.l). Page and Thomas (1977) defined research as a “...systematic investigation to increase knowledge and/or 
understanding” (p.290). One method of systematic qualitative research, referred to as a case study, “is an intensive 
description and analysis of a phenomenon or social unit such as an individual, group, institution, or community” ” 
(Marriam, 1995, p. 108). Similarly, during an authentic assessment process the classroom teacher is involved in 
intensive description and analysis of a student’s performance. Classroom teachers use anecdotal records, checklists, logs, 
journals, portfolios and interviews as authentic assessment data gathering tactics in much the same way that qualitative 
researchers gather data in the field. In a case study methodology, for instance, the primary aim of the systematic 
investigation is to describe the observed situation “in depth, in detail, in context, and holistically” (Patton, 1987, p. 19). 
This is also the goal of a classroom teacher engaged in the authentic assessment process. Parelleling the teacher’s role 
in the classroom with that of a qualitative researcher places a unique perspective on the traditional view of a classroom 
teacher. This unique perspective expands the definition of today’s classroom teachers to that of a “teachcr/researcher”, 
involved in documenting and evaluating key observable educational situations. This, in turn, results in the creation of a 
broader, more genuine picture of student performance and offers more opportunities for the improvement of student 
learning. 

The use of qualitative observational methodology is a key to increasing the effectiveness of the authentic 
assessment process. Through consistent and systematic classroom observation, the teacher is able to gain a deeper 
understanding of how a child learns as well as the end product of the learning experience. Just as in qualitative 
methodology, the “observer is the instrument” (Patton, 1987, p. 12) who carefully examines trends or patterns while 
emphasizing “the importance of the meaning of human behavior and the social-cultural context of social interaction” (ct 
al M 1987, p.20). The integration of qualitative observational methods can provide teachers with a systematic pedagogical 
framework in which to improve the accuracy, validity, and reliability of the authentic assessment process. 

This paper focuses on a systematic method of developing an authentic assessment instrument that incorporates a 
three phase process of what is to be assessed, how it is to be measured, and how to interpret verbal and non-verbal 
messages or data. 

What is to be assessed? 

Prior to the implementation of any form of assessment, a clear understanding of what is to be assessed is 
necessary. This includes a fundamental understanding of the purpose of the assessment process, the learner, and the 
subject matter to be taught. 
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If the primary purpose of the assessment is to "describe the extent to which students have attained particular 
knowledge and skills, your assessment should focus on the outcomes or product of student learning" (Herman, 

Aschbacher, Winters, 1992, p.23). "However, if your purpose is diagnosis and improvement, such as diagnosing a 
student’s strengths and weaknesses, prescribing the most appropriate instructional programs, or identifying strategies 
students use well and those they need help with, you’ll want an assessment that gives you information about the process 
as well as the outcome" (et al., 1992, p.23). 

Understanding and recognizing the uniqueness of learners and their learning styles is an important step in the 
assessment process. The multiple intelligence theory, advanced by Howard Gardner (1983), espoused a shift in 
philosophy in which the assessor recognizes the learner’s framework of intelligence based on a broad range of abilities. 
Fully exploring and supporting the potential of all learners has the potential to maximize the learning experience and 
establish a powerful sense of self-confidence for all learners. 

Finally, a clear understanding of the subject matter is a necessary step in the assessment process. Without a clear 
understanding of the subject matter, determining important learning outcomes would be a difficult, if not impossible, 
task. 

How is it to be measured? 

One method of measuring student progress is through the use of a rubric. A rubric is a rule of procedure. "When 
the word is used in connection with assessment, a rubric is a scoring guide that differentiates... on an articulated scale..." 
(Jasmine, 1993. p.9). There are two kinds of rubrics, holistic and analytic. According to Jasmine (1993), a holistic 
rubric is used to measure the overall effect of a task and is qualitative in nature. An analytic rubric consists of score 
points assigned to various elements and is quantitative in nature. Determining clear and educationally sound learning 
outcomes is a major step in the assessment process. Well-constructed learning outcomes establish a strong foundation 
for prioritizing and creating a properly formulated rubric. 

Setting criteria and standards of a rubric is a main component of authentic assessment. "Perhaps most 
important, scoring criteria make public what is being judged and, in many cases, the standards for acceptable performance, 
it Thus, criteria communicate your goals and achievement stands is" (Herman, Aschbacher, Winters, 1992, p.44). 

Selecting the task to match the authentic assessment is also a crucial point in the assessment process. Aligning 
the assessment task with the intended outcomes should be done to ensure that the assessment can, in fact, measure the 
learning to take place. "When considering assessment tasks, your best choices are those you believe most closely target 
your instructional aims and allow your students to demonstrate their progress and capabilities" (Herman, Aschbacher, 
Winters, 1992, p.33). Herman, Aschbacher, Winters (1992) posed six questions to help educators choose good tasks: 

• Does the task match specific instructional intentions? 

• Does the task adequately represent the content and skills you expect students to attain? 

• Does the task enable students to demonstrate their progress and capabilities? 

• Does the assessment use authentic, real-world tasks 9 

• Does the task lend itself to an interdisciplinary approach? 

• Can the task be structured to provide measures of several goals? (p.35-37) 

Choosing an appropriate task must then be followed by describing the assessment task in a manner that is clear and 
documentable in order for others to replicate and/or interpret the finding. Finally, Herman, Aschbacher, Winters (1993) 
offer a set of criteria that help to ensure that the appropriate learning tasks lead to sound assessment. These criteria are: 

• Do the tasks match the important outcome goals you have set for the students? 

• Do they pose an enduring problem type, the type of problems and situations that students are likely to face 
repeatedly in school and their future lives? 

• Are the tasks fair and free of bias? 

• Will the tasks be credible to important constituencies? 

• Will the tasks be meaningful and engaging to students so that they will be motivated to show their capabilities? 

• Are the tasks instructional^ related/teachable? 

• Are the tasks feasible for implementation in your classroom or school in terms of space, equipment, time, costs, and 
so forth?, (p. 41-43) 



How to interpret verbal & non-verbal messages or data. 

The assessment process is a matter of the verification of student learning. In the authentic assessment process, 
students are engaged in authentic and meaningful tasks. According to Wiggins (1993), the assessment process is 
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authentic when we directly examine student performance on worthy intellectual tasks (p. 3) therefore, the reliance on 
r:ound first-hand observation of both verbal and non-verbal learning situations is a vital component in authentic 
assessment. Teachers involved in authentic assessment need to develop strong observational skills. Harp (1991) 
discusses observation as a powerful tool for the evaluation of students as they are engaged in the performance of real 
world tasks (p. 75). 

First-hand observation provides an understanding of both the activity and the context in which the activity is 
performed. It allows the observer to make more reliable inferences concerning the performance activity. Better 
understanding and more reliable inferences are a direct result of the observer’s ability to concentrate on worthy intellectual 
tasks in order to evaluate the performance activity more clearly. 

The ability to interpret verbal and non-verbal messages accurately, validly, and reliably through the observation 
of learning situations requires particularly careful analysis and judgment on the part of the observer. “The validity and 
reliability of qualitative data depend to a great extent on the methodological skill, sensitivity, and training of the 
evaluator. Systematic and rigorous observation involves far more than just being present and looking around” (Patton, 
1987, p. 8). The value of observational data is developed through the observer’s descriptive documentation of the setting, 
the activities, and meaning of what was observed. 



Critical Viewing 

The teacher has always been a critic to some extent, but are teachers critical viewers as well? According to James 
Brown (1991), "The critic, and the critical viewer, must be grounded in two essential areas of this act of critical 
judgment: the area of facts (what is informational data) and the area of norms or standards (what ought to be criteria)" (p. 
24). Thus, teachers are critical viewers by nature, but perhaps not by systematic process. 

The process of critically viewing performances involving human action has not been adequately addressed through 
research. Perhaps this is because no critical viewing model exists to guide teachers in systematically assessing their 
students. Traditionally, critical viewing has involved the interpretation of verbal and non-verbal information involving 
some form of media such as movies, photographs or television. As Ploghoft and Anderson (1982) stated, "individuals 
trained in critical viewing skills will be equipped with criteria for evaluating intention, motives, and audience response 
(students), for assigning value or worth to the message" (p. 5). This paper attempts to provide a critical viewing model 
for education by providing teachers with a systematic three phase process for the development of an authentic assessment 
instrument which includes a critical viewing component. However, before discussing the third phase that involves 
critical viewing for teacher/researchers, it is important to understand how and where critical viewing skills developed. 

Critical reading skills evolved into critical viewing skills in order to better analyze electronically mediated 
messages. As John Long (1989) observed, "much of the groundwork for critical viewing was derived from work in 
critical reading. In critical reading, individuals learn procedures to extract and assimilate from the print media" (p. 12). 
Critical viewing follows a similar process but involves electronic media. As society and media became more 
sophisticated, the need to understand how information was interpreted through electronic media grew. 

Several projects and studies were conducted to analyze and develop standard critical viewing skills for elementary, 
middle school, high school and adult students. The elementary project was conducted by the Southwest Educational 
Development Laboratory, WNET a public television station in New York published the middle school segment, the Far 
West Laboratory in San Francisco developed the high school study and Boston University's School of Public 
Communications developed the adult program. The project's goal was to assist individuals in monitoring their own 
viewing habits; in other words, to teach people to view television from a perspective other than that of the medium. 

There have been many different sets of skills developed for various mediated situations and targeted at certain age 
levels. For example, the Southwest Educational Development Laboratory defined critical viewing skills as the ability to 
distinguish program elements, make judicious use of viewing time, understand the psychological implications of 
advertising, distinguish fact from fiction, recognize and appreciate differing views, understand content of dramatic 
presentation, understand style of dramatic presentation and understand the relationship between television and the printed 
word (Brown, 1991, p.96). Although a general standard of critical viewing skills is difficult to identify, the majority of 
projects and studies share the concept that people must not be passive consumers of information because they lose the 
ability to understand the world from varied perspectives. Adams & Hamm (1988) observed, "Intelligent consumers of 
video information need to be able to sort out the meaningful from the trivial" (p. 82). An increased level of critical 
awareness has the potential to move the viewer from passive information processing to active information processing. 
This is true of any viewing situation, be it mediated or live. In the age of information, how one views life can help or 
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hinder understanding reality. Thus, critical viewing is more than understanding mediated messages. It is understanding 
ail types of messages from mediated to live. As Adams & Hamm stated, "Effective viewing of real world images requires 
a critical sense for both information and shades of meaning" (1988, p. 82). 

Critical Viewing in an Educational Setting 

Critical viewing of student performance is the next step in the evolution of critical viewing skills. The general 
framework consists of "the principles of intervention, goal attainment, cultural understanding, and literacy" (Long, 1989, 
p 13) Intervention involves understanding the process of perception and may be altered through self-regulatory skills. 

This is a major step because it is at this level that the viewer stops being a passive viewer and becomes an active viewer. 
The active viewer begins to understand how his/her own internal perceptions and biases are developed. Individuals bring a 
set of beliefs to every situation. Discovering how to separate one's own perception from the visual messages in a . 
situation is the key to becoming a critical viewer. An example of this is a teacher using a rubric to assess a child’s 
performance. In order to incorporate critical viewing into the assessment, the teacher must use intervention to understand 
how his/her internal perspectives may affect the assessment. For instance, if a teacher is tired or has had difficulty with a 
particular child in the past, a skewed perspective of the child's ability may occur. This skewed perspective may in turn 

alter the assessment process. . . 

The next step in the process of becoming a critical viewer is goal attainment. Goal attainment involves 

monitoring and understanding one's own reasons for attending to the learning situation as well as recognizing the 
student’s own (and possibly diFerent) rationale for participation. In other words, the viewer and the student performing 
the learning activity each have i set of goals. These goals do not necessarily refer to learning activity goals but refer to 
the personal goals of the teacher and of the student at the time of assessment. The teacher may want the student to 
complete the activity in a certain way and the student may want to just get through the task. It is the responsibility of 
the critical viewer to understand his/her personal goals as well as the goals of the person being assessed. By 
accomplishing this, the critical viewer can ensure that the goals of the learning activity are mutually understood and that 

the.evaluation of the students’ performance is not adversely affected by a clash of personal goals. 

The third step in the critical viewing process is cultural understanding. Cultural understanding is knowing where 
the learning situation exists within the social schema. It is the opposite of the intervention step because the viewer now 
focuses on external awareness of the perceptions and biases of the individuals being assessed. For example, student 
race religion, gender, etc. play a large role in their perception of any learning situation. The students' perceptions and 
biases will be likely to affect their performances and thus adversely alter the assessment unless the teacher/viewer 

incorporates cultural context into the assessment situation. 

The final step in the critical viewing process is literacy. Literacy is understanding the overall grammar of the 
learning situation, such as body language or actions of the learner in the course of an activity (physical, behavioral, 
and/or cognitive). It is at this step in the process that the viewer can take all of the knowledge gained from the first three 
steps of the critical viewing process and actually interpret what is seen during the learning activity. For example, the 
body language, vocal tone and verbal usage of a diabetic child with low blood sugar will appear as a normal child 
behaving poorly. A trained and well-prepared teacher critically viewing the learning situation will assess the whole child 
within the learning context. A teacher passively viewing the situation might assess the child incorrectly by not being 
aware of perceptions and biases, personal and other goals, external awareness of others' perceptions and biases, and the 
literacy of how to communicate. 

The four steps to critical viewing skills create skilled observers who are trained and prepared for authentic 
assessment. Critical viewing adds an important component to the authentic assessment process, filling in valuable 
contextual information about performance, thus creating a more valid and reliable assessment. 

Pre-Assessment Focusi ng Session 

A well -trained and prepared critical viewer recognizes the need for intense focus and concentration in the 
observation process. Prior to entering an observational situation for the purposes of assessment, a classroom teacher 
must set aside time to mentally prepare for the complex exercise of interpreting the verbal and non-verbal messages in a 
student’s performance. This process is referred to in this paper as a pre-assessment focusing session. A pre-assessment 
focusing session utilizes the four steps of critical viewing and is intended to focus a teacher/researcher’s critical viewing 
skills in order to filter out personal perceptions or biases that could be present during an assessment situation. This 
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process involves little time commentment but such focused preparation is essential for effective critical viewing and 
assessment. Once a teacher/researcher becomes familar with the steps of critical viewing the pre-assessment focusing 
session should become an integral part of the assessment process. 

Summary 

What. is taught, how it is taught, and how it is assessed are critical issues within the educational community. 

The three phases discussed in this paper integrate the major steps involved in the development of a sound authentic 
assessment instrument. 

The first phase addresses the need to carefully formulate a clear understanding of What is to be assessed? This 
includes the identification of the purpose of the assessment, recognizing the uniqueness of the learners, and possessing a 
knowledge base in the relevant subject matter. 

The second phase looks closely at How is it to be measured? Determining appropriate learning outcomes will 
drive the instructional and assessment processes, therefore the formulation of the learning outcomes and the criteria and 
standards that will be applied to them are critical components. This leads directly to the selection of a suitable 
assessment task. The assessment task is an essential component of authentic assessment and is not only important from 
the stand point of validation of student learning but is vital for the students themselves to understand the expectations of 
the learning task. Ralph Tyler (1949), father of modem evaluation, commented that without an understanding of what a 
learner is supposed to know or how it is to be expressed, makes determining whether or not learning has actually occurred 
very difficult. 

The third phase describes how to interpret verbal and non-verbal messages or data in the observational process of 
authentic assessment Goodman stated, “teachers are constant kid watchers” (1986, p.41). As such, classroom teachers 
need to be trained and well-prepared observers of all activities throughout the school day. Teachers must remain focused 
and objective just as qualitative researchers in scientific fields do. Trained and well-prepared classroom teachers become 
more astute observers of the underlying meaning and social context of student behavior, thus creating the potential for 
improved quality of their findings and ultimately of their assessment procedures. 

Since teacher preparation is essential for accurate interpretation of verbal and non-verbal messages, the inclusion 
of a brief pre-assessment focusing session prior to an observational assessment will positively impact the assessment 
process. This focusing session sharpens a teacher’s concentration and heightens the teacher’s awareness of personal 
interventions, goals, the cultural context, and literacy issues that may interfere with, contaminate or unfairly influence 
the assessment process. 

The integration of the three phases process described in this paper into the authentic assessment process provides 
the classroom teacher with a systematic method of developing an effective assessment instrument. Processing 
information through the first, second and the third phases gives a classroom teacher a sense of confidence in the 
development of their own authentic assessment instrument as well as their observational skills. 

As classroom teachers strengthen their observational skills and begin to recognize their roles as teacher- researchers 
involved in important qualitative educational research, hidden benefits are likely to be discovered. The impact of these 
hidden benefits will be seen on two levels. On one level, the classroom teacher will experience an increase in confidence 
and professionalism. On another level, there will be direct positive effect on daily instructional and social classroom 
activities. The significance of these powerful hidden benefits will, by their very nature, have a positive impact on 
teachers’ creativity and on their overall effectiveness in the instructional and assessment process. 
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